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[Document Name] Specification 

[Title of the Invention] IMAGE EXTRACTING APPARATUS 

[Scope of Claims for Patent] 

[Claim l] A image extracting apparatus, comprising: an image 
database in which a plurality of images are stored; 

detecting means for detecting a orientation of a subject in an image; 

and 

extracting means for extracting an image including a subject in a 
predetermined orientation from the image database based on the result of 
the detection by the detecting means. 

[Claim 2] A image extracting apparatus, comprising: inputting 
means for inputting a moving image; 

detecting means for detecting a orientation of a subject in the input 
moving image; and 

extracting means for extracting a still image including a subject in a 
predetermined orientation from the input moving image based on the result 
of the detection by the detecting means. 

[Claim 3] A image extracting apparatus, comprising: shooting 
means for shooting a subject; 

detecting means for detecting a orientation of a subject in an image 
shot by the shooting means; 

storing means for storing a image; and 

controlling means for making storing means to store a image when 
the detecting means detects a orientation of a subject in a predetermined 
orientation. 
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[Claim 4] The image extracting apparatus according to any of 
claims 1-3, further comprising^ specifying means for specifying a orientation 
of a subject in the image. 

[Claim 5] The image extracting apparatus according to any of 
claims 1*3, wherein the detecting means detects a orientation of a plurality 
of subjects in an image respectively and further comprises counting means 
for counting a number of subjects in the predetermined orientation. 

[Claim 6] The image extracting apparatus according to claim 3, 
wherein the shooting means shoots a subject at predetermined interval. 
[Detailed Description of the Invention] 

[Technical Field to Which the Invention Belongs] 

The present invention relates to an image extracting apparatus and 
particularly relates to an image extracting apparatus extracting desired 
image from a plurality of images. 

[Prior Art] 

Conventionally, as cameras for taking pictures, film-based cameras 
and digital still cameras have become widespread. With these cameras, a 
subject in a desired pose can be shot by situating the subject within the 
frame and depressing the shutter button at the moment when the subject 
strikes the desired pose. 

[Problems to be Solved by the Invention] 

In taking a picture, when the subject is stationary, it is necessary to 
pay attention only to situating the stationary subject within the frame in 
good balance. However, when the subject is moving, it is necessary to 
depress the shutter button at the moment when the moving subject strikes a 
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desired pose or makes a desired facial expression. 

However, it is sometimes difficult to depress the shutter button at 
the moment when the moving subject strikes a predetermined pose or 
makes a predetermined facial expression. For example, when the subject is 
an infant, problems arise such that when shooting is performed at a small 
distance from the infant, his natural facial expression cannot be shot 
because he is conscious of the camera and that the user misses depressing 
the shutter button at the moment when the infant turns his eyes on the 
camera because the infant is moving fast and the user's attention is 
therefore directed to situating the infant within the frame. 

When a group photo is taken with two or more children as the 
subjects, it is difficult to make all the children to strike a pose facing toward 
the front, and it is difficult to depress the shutter button at the moment 
when all the children face toward the front. 

The present invention has been made to eliminate above problems, 
and one of its object is to provide an image extracting apparatus capable of 
obtaining an image in which the subject is in a desired pose. 

Another object of the present invention is to provide an image 
extracting apparatus capable of obtaining from a moving image a still image 
in which the subject is in a desired pose. 

Still another object of the present invention is to provide an image 
extracting apparatus preventing a good moment for picturization from being 
missed to obtain an image in which the subject is in a desired pose. 

Still another object of the present invention is to provide an image 
extracting apparatus capable of obtaining an image in which a 
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predetermined number of subjects are in a desired pose when a plurality of 
subjects is shot. 

[Means for Solving the Problems] 

To achieve the above -described object, a image extracting apparatus 
according to one aspect of the present invention includes an image database 
in which a plurality of images are stored, detecting means for detecting a 
orientation of a subject in an image, and extracting means for extracting an 
image including a subject in a predetermined orientation from the image 
database based on the result of the detection by the detecting means. 

According to the invention, a plurality of images are stored in an 
image database and a orientation of a subject in an image is detected and an 
image including a subject in a predetermined orientation is extracted from 
the image database based on the result of the detection. Accordingly, it is 
possible to provide an image extracting apparatus capable of obtaining an 
image in which a subject is in a desired pose. 

According to another aspect of the invention, a image extracting 
apparatus includes inputting means for inputting a moving image, detecting 
means for detecting a orientation of a subject in the input moving image and 
extracting means for extracting a still image including a subject in a 
predetermined orientation from the input moving image based on the result 
of the detection by the detecting means. 

According to the invention, a orientation of a subject in the input 
moving image is detected and a still image including a subject in a 
predetermined orientation is extracted from the input moving image based 
on the result of the detection. Accordingly, it is possible to provide an image 
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extracting apparatus capable of obtaining a still image in which a subject is 
in a desired pose from the input moving image. 

According to still another aspect of the invention, a image extracting 
apparatus includes shooting means for shooting a subject, detecting means 
for detecting a orientation of a subject in an image shot by the shooting 
means, storing means for storing a image and controlling means for making 
storing means to store a image when the detecting means detects a 
orientation of a subject in a predetermined orientation. 

According to the invention, a orientation of a subject in the shot 
image is detected and a image is stored when the detecting means detects 
the orientation of a subject in a predetermined orientation. Accordingly, it is 
possible to provide an image extracting apparatus preventing a good 
moment for picturization from being missed to obtain an image in which the 
subject is in a desired pose. 

Preferably, the image extracting apparatus further includes 
specifying means for specifying a orientation of a subject in a image. 

According to the invention, it is possible to specify a orientation of a 
subject in the image, so it is possible to extract the image including a subject 
in a desired orientation. 

Preferably, the detecting means of the image extracting apparatus 
includes counting means for detecting orientation of a plurahty of subjects 
in an image respectively and counting a number of subjects in the 
predetermined orientation. 

According to the invention, orientation of a plurality of subjects in an 
image is detected respectively and a number of subjects in the 
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predetermined orientation is counted. Accordingly, it is possible to provide 
an image extracting apparatus capable of obtaining a image in which 
predetermined number of subjects are in a desired pose when a plurality of 
subjects is shot. 

Preferably, a shooting means of the image extracting apparatus is 
featured by shooting a subject at predetermined interval. 

According to the invention, a subject is shot at predetermined 
interval, so it is possible to prevent a good moment from being missed to the 
subject is in a desired pose. 

[Embodiment] 

Hereinafter, embodiments of the present invention will be described 
with reference to the drawings. In the figures, like reference numbers 
represent like or corresponding members, and descriptions thereof will not 
be repeated. 

[First Embodiment] 

FIG. 1 is a rear view of a camera of a first embodiment of the present 
invention. The camera 100 of this embodiment includes a shutter button 2, 
a Uquid crystal display 4 for displaying various pieces of information, a 
viewfinder 3, a function switching button 5, an orientation specifying button 
6, and a number-of-persons setting button 7. 

The function switching button 5 is a button switch for switching 
functions of the camera such as switching between use and nonuse of the 
flash and switching between use and nonuse of the self-timer. Switching 
between a normal shooting mode and a face picturizing mode can be made 
by use of the function switching button 5. In the normal shooting mode, 
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the subject is shot in response to a depression of the shutter button 2. In 
the face picturizing mode, although details thereof will be described later, 
the subject is picturized on condition that the subject strikes a desired pose. 
In addition, in the face picturizing mode, a number-of-frames priority mode 
or a time priority mode can be specified. This specification is also 
performed by the function switching button 5. 

The orientation specifying button 6 is a button switch for setting the 
pose of the subject, particularly, the orientation of the subject's face when 
the camera 100 is in the face picturizing mode. The orientation of the 
subject's face is determined by the relative position relationship between the 
subject's face and the camera 100. More specifically, the orientation of the 
subject's face is determined by the direction from the subject to the camera 
such as when the subject faces toward the camera 100 or when the subject's 
head is turned 90"* leftward or rightward from the direction to the camera 
100. The angle of the face of the subject image included in the picturized 
image can be specified by the orientation specifying button 6. 

The liquid crystal display 4 displays the image to be picturized, 
when the camera 100 is in the face picturizing mode. With this, the 
subject's pose can be confirmed by viewing the liquid crystal display 4. The 
liquid crystal display 4 also displays various setting conditions such as the 
remaining number of frames and the set shooting mode, the current time 
and the like. While the liquid crystal display 4 is provided on the back of 
the camera 100, display may be provided within the viewfinder 3. 

FIG. 2 is a function block diagram showing functions of the camera 
100 of the first embodiment. Referring to FIG. 2, the camera 100 of this 
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embodiment includes an image inputting portion 101 for inputting images, a 
specified orientation face detecting portion 102 for detecting the orientations 
of the faces of the subject images included in the input images, an image 
selecting portion 103 for selecting an image including a subject image that is 
in a predetermined orientation from among the input images, and an image 
outputting portion 104 for outputting the selected image. 

The image inputting portion 101 is a light receiving device such as a 
charge coupled device (CCD), and outputs image data to the specified 
orientation face detecting portion 102 and the image selecting portion 103. 
The specified orientation face detecting portion 102 detects a subject image 
that is in a specified orientation by detecting the orientations of the faces of 
the subject images included in the images received from the image inputting 
portion 101. Here, the specified orientation is determined by the relative 
positions of the subject and the camera^ a frontal face when the subject faces 
toward the camera; a face in full profile when the subject turns his head 90 
leftward or rightward from the camera; and a face in half profile when the 
subject turns his head 45 ^ leftward or rightward from the camera. The 
orientation is specified by the orientation specifying button 6 as an angle 
through which the subject turns his head from the camera. 

The specified orientation face detecting portion 102 detects a face 
that is in the specified orientation predetermined by the orientation 
specifying button 6. When the subject in an image is in the specified 
orientation, a predetermined signal is output to the image selecting portion 
103, so that the image picturized by the image inputting portion 101 is 
selected and output to the image outputting portion 104. The image 
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outputting portion 104 may be the liquid crystal display 4, or may be a 
semiconductor memory. Moreover, the image outputting portion 104 may 
be an apparatus that exposes silver hahde film. 

Next, a picturization processing performed when the camera 100 is 
in a face selecting mode will be described. The camera 100 is placed in the 
face selecting mode by depressing the function switching button 5. FIG. 3 
is a flowchart of the picturization processing performed when the camera 
100 of the first embodiment is in the face selecting mode. Referring to FIG. 
3, first, the camera 100 is placed in the face selecting mode (step Si). The 
setting of the face selecting mode is performed by depressing the function 
switching button 5. 

Then, a face orientation is specified (step S2). A display prompting 
the user to specify a face orientation is provided on the liquid crystal 4, and 
a face orientation is specified by depressing the orientation specifying 
button 6. The face orientation specified at this step is the specified 
orientation. The specified orientation is specified by the orientation 
specifying button 6 as an angle through which the subject turns his head 
from the camera. For example, the specified orientation is changed to the 
fi-ont, 45° rightward, 90° rightward, 45° leftward and 90° leftward in this 
order every time the orientation specifying button 6 is depressed, and is 
selected. 

Then, a face that is in the specified orientation is searched for (step 
S3). Whether a subject image whose face is in the orientation specified at 
step S2 is included in images or not is determined. This will be concretely 
described later. 
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Then, whether a subject image whose face is in the specified 
orientation is detected or not is determined (step S4). When such a subject 
image is detected, the process proceeds to step S5. When a subject image 
whose face is in the specified orientation is not detected, the process 
proceeds to step S3 to repeat the processing until a subject image whose face 
is in the specified orientation is detected. 

At step S5, the image picturized by the image inputting portion 101 
is selected by the image selecting portion 103 and output to the image 
outputting portion 104. By this, the picturization processing is performed. 

Then, whether the face selecting mode is canceled or not is 
determined (step S5). When it is canceled, the processing is ended, and 
when it is not canceled, the process proceeds to step S2. 

The face selecting mode comprises two modes, the number-of-frames 
priority mode and the time priority mode. In the number-of frames priority 
mode, the face selecting mode is canceled on condition that a predetermined 
number of images have been selected by the image selecting portion 103. 
In the time priority mode, the face selecting mode is canceled on condition 
that a predetermined time has elapsed since the camera 100 was placed in 
the face selecting mode. In the time priority mode, by step S6 being an 
interrupt processing, there can be cases where no image is selected by the 
image selecting portion 103 even when a subject image whose face is in the 
specified orientation is not detected at step S4. 

Next, techniques for detecting a face in a predetermined orientation 
from images will be described. There are two techniques for detecting the 
orientation of the face of a subject image included in an imaged a technique 
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using neural networks; and a technique using a probability model. The 
technique using neural networks is described in H. A. Rowley, "Neutral 
Network-Based Learning for View-Based Human Face Detection," 
CMU-CS-99-117, 1999. The technique using a probability model is 
described in H. Schneiderman, and T. Kanade, "Probabilistic Modeling of 
Local Appearance and Spatial Relationships for Object Recognition," 
CVPR98. In the camera 100 of this embodiment, either of these techniques 
may be employed. 

[Technique Using Neutral Networks] 

FIG. 4 is a view of assistance in explaining the principle of the 
technique using neural networks. The technique using neural networks 
uses two kinds of networks, a pose estimation network and a face detection 
network. The face detection network is provided for each of a plurality of 
categories into which faces are classified based on the angles through which 
the faces are turned in the horizontal direction from the front. The 
categories are called face orientation categories. 

Here, the front indicates a case where the subject faces toward the 
camera 100. The face orientation categories are classified based on how 
much the subject's face is turned in the horizontal direction from the front. 
The face orientation categories are categories of the face detection network. 

In FIG. 4, a face detection network 206A is a category when the 
subject's face looks toward the front (frontal category), a face detection 
network 206A is a category when the subject's face is turned 45*^ rightward 
from the front (45*^ rightward category), and a face detection network 206C 
is a category when the subject's face is turned 90** rightward from the front 
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(90° rightward category). Like this, the face detection networks 206A, 
206B and 20C are provided for the categories, respectively. 

The pose estimation network outputs, for an input image, the angle 
of rotation within the plane of the image from an upright orientation, and 
the category of the face detection network. The plane of the image from an 
upright orientation is a plane whose normal is a straight Une connecting the 
camera and the subject. The angle of rotation within the plane of the 
image is the angle of rotation about the normal. Therefore, the angle of 
rotation within the plane of the image from an upright orientation is, for 
example in a case where the subject is a person, the angle of rotation of the 
face when the subject tilts his head. 

FIG. 5 is a flowchart of a processing to determine the orientations of 
faces in an image. The technique using neural networks will be described 
with reference to the principle view of FIG. 4 and the flowchart of FIG. 5. 
Referring to FIGs. 4 and 5, partial images 201, 201A and 201B including 
face images are cut out from an input image 200 (step SIX). Here, an 
example is shown in which the partial image 201 includes a frontal face 
image, the partial image 201A includes a face image turned 45'' rightward, 
and the partial image 201C includes a face image turned 90° rightward. 
The cut out partial images 201, 201Aand 201B are input to pose estimation 
networks 202, 202Aand 200B (step S12). 

The pose estimation networks 202, 202A and 200B output, for the 
face images included in the input partial images 201, 201A and 201B, the 
angles of rotation within the plane of the image from an upright orientation 
and the categories of the face detection network. Then, by use of the angles 
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of rotation within the plane of the image from an upright orientation 203, 
203A and 203B detected by the pose estimation networks 202, 202A and 
202B, the face images included in the partial images 201, 201 A and 20 IB 
are angle-corrected so as to be in an upright orientation (step S13). 
Consequently, the rotation within the plane of the image from an upright 
orientation is corrected, so that the orientations of the faces can be 
determined more accurately. 

Then, corrected partial images 204, 204A and 204B are input to the 
face detection networks 206A, 206B and 206C (step S14). 

This is for the following reason: When the partial images 204, 204A 
and 204B are input to the face detection networks 206A, 206B and 206C, 
the orientations of the face images included in the partial images 204, 204A 
and 204B are unknown. Therefore, at this point of time, the partial images 
204, 204A and 204B are input to the provided face detection networks of all 
the orientations to determine the orientations of the faces in the partial 
images 204, 204A and 204B. 

In each of the face detection networks 206, 206A and 206B, whether 
the subject image included in the input partial image is a face or not is 
determined (steps S15, S16 and S17). Consequently, the orientation of the 
face detection network where the subject image is determined to be a face is 
determined to be the orientation of the face included in the partial image 
(step S18). For example, when an image of a frontal face is input to the 
frontal face detection network, the input image is determined to be a face, 
and the face in the image is determined to be a frontal face. On the 
contrary, when a face turned 45** is input to the frontal face detection 
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network, the face is not determined to be a face. 

When the result of the output from the face detection network is 
such that the subject image is not a face, it is determined that the subject 
image included in the partial image 201, 201Aor 201B is not a face image or 
that the orientation is erroneous. In this manner, the orientations of the 
faces of the subject images included in the partial images of the input image 
200 are obtained. 

[Technique Using Probability Model] 

The technique using a probability model is to determine whether a 
given region is a face in a specified orientation or not by comparing the 
posterior probability that the region is a class of a face in the specified 
orientation and the posterior probability that the region is not the class of a 
face in the specified direction. In actuality, by use of the Bayes rule, the 
ratio between the likelihoods of two classes is calculated, and thresholding 
is performed. The posterior probabilities of the classes are simplified 
starting with a general form, and are converted into a form of a probabiUty 
model having a calculable multiplicity. 

FIG. 6 is a view of assistance in explaining the principle of a 
learning processing in the technique using a probability model. Referring 
to FIG. 6, an input image 300 for learning is divided into overlapping partial 
images 301, and the connection rate distribution of the positions and the 
texture features of the partial images 301 are calculated for each category. 
The texture feature is expressed in a compressed and quantized data form, 
and the probability is calculated by counting the frequency of occurrence of a 
specific texture in a specific position in the entire area of the input image 
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300 for learning. 

The learning processing is a processing to obtain the probabiUty that 
a specific texture occurs in a specific position for both a case where the input 
image 300 includes a face in a specified orientation and a case where the 
input image 300 does not include a face in the specified orientation, that is, 
a processing to count the frequency of occurrence of a specific texture 
feature in the positions of the partial images 301 in the input image 300 
that has been found to include a face, and previously obtain the probabiUty 
thereof. 

Learning is performed in a similar manner when the input image 
300 does not include a face in the specified orientation. The texture feature 
of each partial image 301 undergoes dimensional compression by a principal 
component analysis, and is expressed in a compressed and quantized form 
projected on a lowlevel space. 

FIG. 7 is a flowchart of the learning processing. Referring to FIGs. 
6 and 7, the overlapping partial images 301 are cut out from the input 
image 300 (step S21). 

Then, a specific texture feature is created (step S22). Then, the 
frequency of occurrence of the texture feature in the position of a partial 
image 301 in the input image 300 is counted (step S23). 

Then, whether it is possible to move to the next partial image 301 or 
not is determined (step S24), and when it is possible, the process moves to 
the next region (step S25). A case where it is possible to move to the next 
region is a case where a partial image 301 to be processed is present, and a 
case where it is impossible to move to the next region is a case where no 
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partial image 301 to be processed is present. When it is impossible to move 
to the next region, a connection rate table of the region and the texture is 
generated (step S26). The connection rate table thus obtained is used for a 
recognition processing described next. 

FIG. 8 is a schematic view of assistance in explaining the principle of 
the recognition processing. FIG. 9 is a flowchart of the recognition 
processing. Referring to FIGs. 8 and 9, a window region 311 is set in an 
input image 310 (step S3l). The window region is a region included in the 
input image 310, and can be cut out arbitrarily. 

Then, overlapping partial images 312 are cut out in the window 
region 311 (step S32). Then, a texture feature in a partial image 312 is 
created (step S33), and from the created texture feature and the position of 
the partial image in the window region 311, the probability that the face is 
in a specific orientation is obtained by use of the connection rate table 
created in the learning processing (step S34). 

Then, whether it is possible to move to the next partial image or not 
is determined (step S35). When it is possible, the probability obtained at 
step S34 is stored, and after moving to the next partial image (step S36), the 
process proceeds to step S32. 

When there is no partial image to which the process is to move, the 
product of the probabilities of all the partial images is obtained (step S37). 
This product is set as the likeUhood of the window region 311. Then, from 
the obtained likehhood, whether a face in the specified orientation is 
present in the window region or not is determined (step S38). 

Then, whether a window region to be set next is present in the input 
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image 300 or not is determined (step S39). When a window region is 
present, the process proceeds to step S40, and when no window region is 
present, the process is ended. At step S40, the process moves to the next 
window region, and proceeds to step S31. 

As described above, in the camera 100 of this embodiment, since 
picturization is performed when the face included in the input image is in a 
predetermined orientation, when a portrait of an infant or the like is taken, 
a natural facial expression of an infant whose face is in a desired orientation 
can be shot without a good moment to depress the shutter button being 
missed. 

Moreover, since the user can follow the subject with attention only to 
situating the subject within the viewfinder in good balance, the subject can 
be shot with ease, and further, it is only when the subject is in a desired 
orientation that the shutter button is depressed. Consequently, even when 
the subject is moving fast, the subject can be shot without a good moment to 
depress the shutter button being missed. 

[Second Embodiment] 

FIG. 10 is a function block diagram of assistance in explaining 
functions of a camera 150 of the second embodiment. Referring to FIG. 10, 
the camera 150 of the second embodiment includes an image inputting 
portion 101 for inputting images, a specified orientation face detecting 
portion 102 connected to the image inputting portion and for detecting faces 
in a specified orientation from images included in an input image, a 
specified orientation face counting portion 106 for counting the number of 
faces in the specified orientation, an image selecting portion 103 for 
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selecting the image input by the image inputting portion 101 when a 
predetermined number of faces are in the specified orientation, a 
number-of-persons setting portion 105 for inputting a condition for selecting 
an image by the image selecting portion 103, and an image outputting 
portion 104 for outputting the selected image. 

The camera 150 of the second embodiment is the camera 100 of the 
first embodiment to which the specified orientation face counting portion 
106 and the number of-persons counting portion 105 are added. The 
structure except this is similar to that of the camera 100 of the first 
embodiment, and therefore, the description thereof will not be repeated. 

The number-of-persons setting portion 105 includes a 
number-of-persons setting button 7 and a part of the liquid crystal display 4. 
On the part of the liquid crystal display 4, a number in accordance with the 
number of depressions of the number-of-persons setting button 7 is 
displayed. The number displayed on the Uquid crystal display 4 is set as 
the number of persons. The specified orientation face counting portion 106 
counts the number of faces detected to be in the specified orientation by the 
specified orientation face detecting portion 102. 

FIG. 11 is a flowchart of a picturization processing performed when 
the camera 150 of the second embodiment is in the face selecting mode. 
When switching to the face selecting mode is made by depressing the 
function switching button 5, the camera 150 is placed in the face selecting 
mode (step S51). 

When the camera 150 is placed in the face selecting mode, a region 
for displaying the number of persons is provided in a part of the liquid 
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crystal display 4. Then, the number of persons can be set by depressing the 
number of-persons button 7 (step S52). Then, by depressing the shutter 
button 2, the number displayed on the liquid crystal display 4 at that point 
of time is set as the number of persons. In this embodiment, the specified 
orientation is toward the front. The camera 150 may be structured so that 
the specified orientation is not Hmited to toward the front but various 
orientations can be specified. 

At the next step S53, faces in the specified orientation, that is, 
looking toward the camera are searched for (step S53). At step S54, the 
number of faces looking toward the front among the faces included in the 
image is counted by the specified orientation face counting portion 106 (step 
S54). Then, it is determined whether or not the number of faces counted by 
the specified orientation face counting portion 106 is equal to or larger than 
the number of persons set by the number-of-persons setting portion 105 
(step S55). When the counted number is equal to or larger than the set 
number of persons, the process proceeds to step S56, where the image is 
selected and output to the image outputting portion 104. When the 
counted number is smaller than the set number of persons, the process 
proceeds to step S53 to repeat the above-described processing on an image 
newly input from the image inputting portion 101. 

At step S57, whether the face selecting mode (frontal face selecting 
mode) is canceled or not is determined. When it is canceled, the processing 
is ended, and when it is not canceled, the process proceeds to step S52 to 
repeat the above-described processing. The face selecting mode is canceled 
on condition that a predetermined time has elapsed or that shooting has 
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been performed a predetermined number of times. 

As described above, the camera 150 of the second embodiment is 
effective when the self-timer function is used in taking a group photo on an 
occasion such as a group trip. That is, shooting is performed by the camera 
when the user himself joins the group to be shot and of the persons included 
in the group, a specified number of persons face toward the front. By doing 
this, shooting can be performed without concern for time, although shooting 
is not performed until a predetermined time has elapsed according to the 
self-timer function, . 

Moreover, in a case where the user himself takes a picture with a 
camera, when the subject is a plurality of persons, a picture in which all the 
persons face toward the front can be taken by observing all the persons' 
poses and paying attention only to situating the subjects within the 
viewfinder in good balance when a picture in which all the persons face 
toward the front is taken. 

While an example in which the present invention is appUed to a 
shooting apparatus such as a camera is shown in this embodiment, the 
present invention is applicable to a case where a still image is extracted 
from a video-shot moving image. 

Moreover, it is considered to use the present invention for image 
search of an image database or the like. For example, it is possible to 
extract an image including a person whose head is turned through a specific 
angle or to perform an image search with an image including three persons 
in a specific orientation with the keyword. 

[Third Embodiment] 
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Next, an image extracting apparatus of the third embodiment of the 
present invention will be described. FIG. 12 is a block diagram 
schematically showing the structure of the image extracting apparatus 400 
of the third embodiment. Referring to FIG. 12, the image extracting 
apparatus 400 includes a central processing unit (CPU) 401 for controlling 
the entire image extracting apparatus 400, a keyboard 402, a pointing 
device 402a, a display portion 403, a memory 404, and a storage device 405. 

The pointing device 402a which is used together with a mouse is 
used for inputting data or inputting a search criterion for image search to 
the image extracting apparatus 400. 

The display portion 403 which is a cathode ray tube (CRT) display or 
a liquid crystal display displays a user interface for drawing a screen 
rendered a search criterion, image data rendered a search result, and the 
like. 

In the memory 404, programs to be executed by the CPU 401 and 
data necessary for executing the programs are stored. In the storage device 
405, an image database containing a pluraUty of image data to be searched 
is stored. 

Next, a processing performed by the image extracting apparatus will 
be briefly described. FIG. 13 is a flowchart of the processing performed by 
the image extracting apparatus of the third embodiment. In this example, 
a case will be described where an image including a face in a certain 
orientation, for example, an image including a face looking toward the front 
is to be retrieved. 

Referring to FIG. 13, the user specifies a face search menu with the 



- 21 - 



keyboard 402 and the pointing device 402a of the image extracting 
apparatus 400 (step SlOl). The orientation of the face to be retrieved is set 
from the face search menu (step S102), 

Whether the search has been performed on all the images in the 
image database stored in the storage device 405 or not is determined (step 
S103). When the search has been performed on all the images, the process 
proceeds to step S107, and when the search has not been performed on all 
the images, the process proceeds to step SI 04. 

At step S104, a face in the specified orientation is searched for. For 
the search for a face in the specified orientation, the above-described 
technique using neural networks or technique using a probability model can 
be used. Then, whether a face in the specified orientation is present or not 
is determined (step S105). When such a face is present, the process 
proceeds to step S106, and when such a face is absent, the process returns to 
step S103. 

At step S106, the image including a face in the specified orientation 
is added to the search result list as an image meeting the search criterion. 
Then, the process returns to step S103 to continue the search on all the 
images in the image database. 

When it is determined at step S103 that the search has been 
performed on all the images in the image database, at step S107, the search 
result is displayed in image table or hst form on the display portion 403, and 
the processing is ended. 

FIG. 14 is a flowchart of another processing performed by the image 
extracting apparatus of the third embodiment. In this example, a case will 
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be described where an image including a specified number of faces in a 
specified orientation is to be retrieved when a plurality of faces is included 
in an image. 

Referring to FIG. 14, the user specifies a face search menu with the 
keyboard 402 and the pointing device 402a of the image extracting 
apparatus 400 (step S201). By specifying the orientation of the faces to be 
retrieved and the number of faces in the orientation from the face search 
menu, the orientation and the number are set (step S202), 

Whether the search has been performed on all the images in the 
image database stored in the storage device 405 or not is determined (step 
S203). When the search has been performed on all the images, the process 
proceeds to step S208, and when the search has not been performed on all 
the images, the process proceeds to step S204. 

At step S204, faces in the specified orientation are searched for. 
For the search for faces in the specified orientation, the above-described 
technique using neural networks or technique using a probability model can 
be used. Then, whether faces in the specified orientation are present or not 
is determined (step S205). When such faces are present, the process 
proceeds to step S206, and when such faces are absent, the process returns 
to step S203. 

At step S206, whether or not the number of faces in the specified 
orientation is equal to or larger than the number set at step S202 is 
determined. When the number of faces is equal to or larger than the set 
number, the process proceeds to step S207, and when the number effaces is 
smaller, the process proceeds to step S203. 
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At step S207, the image is added to the search result Ust as an 
image meeting the search criterion. Then, the process returns to step S203 
to continue the search on all the images in the image database. 

When it is determined at step S203 that the search has been 
performed on all the images in the image database, at step S208, the search 
result is displayed in image table or list form on the display portion 403, and 
the processing is ended. 

As described above, with the image extracting apparatus 400 of the 
third embodiment, an image including a face in a predetermined orientation 
can be extracted from among the images stored in the image database. 
When a plurality of faces is included in an image, an image including a 
predetermined number of faces in a predetermined orientation can be 
extracted. 

Moreover, by the storage device 405 of FIG. 12 being a recording and 
reproducing apparatus such as a video tape recorder, a desired image can be 
extracted from among the images recorded on the video tape recorder. 

Although the present invention has been described and illustrated in 
detail, it is clearly understood that the same is by way of illustration and 
example only and is not to be taken by way of limitation, the spirit and 
scope of the present invention being limited only by the terms of the 
appended claims. 

[Brief Description of the Drawings] 

FIG. 1 is a rear view of a camera 100 of a first embodiment of the 
present invention; 

FIG. 2 is a function block diagram showing functions of the camera 
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100 of the first embodiment; 

FIG. 3 is a flowchart of a picturization processing performed when 
the camera of the first embodiment is in a face selecting mode>* 

FIG. 4 is a view of assistance in explaining the principle of a 
technique using neural networks! 

FIG. 5 is a flowchart of a processing to determine the orientations of 
faces in an image; 

FIG. 6 is a view of assistance in explaining the principle of a 
learning processing in a technique using a probability model; 

FIG. 7 is a flowchart of the learning processing; 

FIG. 8 is a schematic view of assistance in explaining the principle of 
a recognition processing; 

FIG. 9 is a flowchart of the recognition processing; 

FIG. 10 is a function block diagram showing functions of a camera of 
a second embodiment; 

FIG. 11 is a flowchart of a picturization processing performed when 
a camera 150 of the second embodiment is in the face selecting mode; 

FIG. 12 is a block diagram schematically showing the structure of an 
image extracting apparatus 400 of a third embodiment; 

FIG. 13 is a flowchart of a processing performed by the image 
extracting apparatus of the third embodiment; and 

FIG. 14 is a flowchart of another processing performed by the image 
extracting apparatus of the third embodiment. 

[Description of the Reference Characters] 

2: shutter button, 3: finder, 4' liquid crystal display, 5: function 
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switching button, 6^ orientation specifying button, l'- number-of-persons 
setting button, 100,150^ camera, 101: image inputting portion, 102: specified 
orientation face detecting portion, 103: image selecting portion, 104: image 
outputting portion, 105: number-of-persons setting portion, 106: specified 
orientation face counting portion 
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Fig. 1 

4 Face Selecting Mode 

Orientation : 45° Rightward 

Number of frames : 
Time : 30 sec 

Fig.2 

101 Image inputting portion 

102 Specified orientation face detecting portion 

103 Image selecting portion 

104 Image outputting portion 

Fig.3 

Start 

51 Set a face selecting mode 

52 Specify a face orientation 

53 Search for a face in the specified orientation 

54 Is a face in the specified orientation detected? 

55 Picturize the image 

56 Is the face selecting mode canceled? 
End 

Fig.4 

206A Front 
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206B 45° rightward 
206C 90° rightward 



Fig.5 

Start 

511 Cut out a image 

512 Pose estimating network 

513 Angle correction using the detected orientation 

514 Input to a face detection network provided for each orientation of 
face respectively 

515 Front face detecting network 

516 45° rightward face detecting network 

517 90° rightward face detecting network 

518 Determine the orientation of face 
Return 

Fig.7 

Learning process 

521 Cut out partial region 

522 Create a texture feature 

523 Count a occurrence of the texture feature in the position of partial 
region 

524 Is it possible to move to the next region? 

525 Move to the next region 

526 Generate a connection rate table of the region and the texture 



- 28 - 



Return 



Fig.9 

Recognition process 

531 Set window region 

532 Cut out partial region in the window region 

533 Create texture feature 

534 Obtain a probability that the face is in a specified orientation from 
the connection rate table made of learning by a position of the partial region 
and the texture feature 

535 Is it possible to move to the next region? 

536 Store the probabiUty and move to a next region 

537 Obtain product of the probabiUties of all partial region 

538 Determine whether a face is in the specified orientation or not 

539 Is it possible to move to a next window region? 

540 Move to the next window region 
Return 



Fig. 10 

101 Image inputting portion 

102 Specified orientation face detecting portion 

103 Image selecting portion 

104 Image outputting portion 

105 Number of persons setting portion 

106 Specified orientation face counting portion 
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Fig. 11 

Start 

551 Set a face selecting mode 

552 Set number of persons 

553 Search for a face in the specified orientation 

554 Count a number of faces 

555 Is the number of face equal to or larger than the set number of 
persons? 

556 Picturize the image 

557 Is the front face selecting mode canceled? 
End 

Fig. 12 

401 CPU 

402 Keyboard 

403 Display 

404 Memory 

405 Storage device 

Fig.13 

Start 

5101 Select a orientation of face search menu 

5102 Specify a orientation of face from the menu 

5103 Is the all search in a DB finished? 
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5104 Search for a face in the specified orientation 

5105 Is a face in the specified orientation presented? 

5106 Add to a search result list 

5107 Display the search result 
End 

Fig. 14 

Start 

5201 Select a orientation of face search menu 

5202 Specify a orientation and number of face from the menu 

5203 Is the all search in a DB finished? 

5204 Search for a face in the specified orientation 

5205 Is a face in the specified orientation presented? 

5206 Number of face in the specified orientation ^ Set number ? 

5207 Add to a search result list 

5208 Display the search result 
End 
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[Document Name] Abstract 
[Abstract] 

[Subject] To obtain a image including a object is in a desired pose. 

[Solving Means] An image database in which a plurality of images is 

stored, detecting means (S104) for detecting the orientation of a subject in 
an image, and extracting means (S106) for extracting an image including a 
subject in a predetermined orientation (S102) from the image database 
based on the result of the detection by the detecting means, are provided. 
[Selected Drawing] Fig. 13 
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